Probability Distributions and Maximum Entropy

If we want to assign probabilities to an event, and see no reason for one outcome to occur more often than any other, then the events are assigned equal probabilities. This is called the principle of insufficient reason, or principle of indifference, and goes back to Laplace. If we happen to know (or learn) something about the non-uniformity of the outcomes, how should the assignment of probabilities be changed? There is an extension of the principle of insufficient reason that suggests what to do. It is called the principle of maximum entropy. After defining entropy and computing it in some examples, we will describe this principle and see how it provides a natural conceptual role for many standard probability distributions (normal, exponential, Laplace, Bernoulli). In particular, the normal distribution will be seen to have a distinguishing property among all continuous probability distributions on R that may be simpler for students in an undergraduate probability course to appreciate than the special role of the normal distribution in the central limit theorem. This paper is organized as follows. In Sections 2 and 3, we describe the principle of maximum entropy in three basic examples. The explanation of these examples is given in Section 4 as a consequence of a general result (Theorem 4.3). Section 5 provides further illustrations of the maximum entropy principle, with details largely left to the reader as practice. In Section 6 we state Shannon’s theorem, which characterizes the entropy function on finite sample spaces. Finally, in Section 7, we prove a theorem about positivity of maximum entropy distributions in the presence of suitable constraints, and derive a uniqueness theorem. In that final section entropy is considered on abstract measure spaces, while the earlier part is written in the language of discrete and continuous probability distributions in order to be more accessible to a motivated undergraduate studying probability.